The employment of higher-order modulation in terabit optical fiber transmission systems may increase spectral efficiency and make better use of transmission capacity of fiber infrastructure. However, transmission signal quality and transmission channel impairments may affect performance of higher-order modulation systems. Advances in high speed analog-to-digital converters (ADCs) and deep submicron semiconductor processing technologies in digital signal processors (DSPs) may enable advanced digital signal processing techniques to be applied for signal conditioning and/or distortion compensation at optical transmitters and/or receivers.